[SPARK-60716][CONNECT] Convert explicit pandas date columns through Arrow - #138

…schemas

…ations

…umPy frames

…ow CI suite

…mantics

Use explicit LongType schema names when constructing Arrow columns. Limit the
columnar path to concrete pandas/NumPy arrays and native schema descriptors,
including authentic converter aliases retained by already imported modules.
Keep nullable values in non-nullable fields on the native row path so local
validation still precedes configuration RPCs. Support warm Arrow importers that
resolve converters locally.

Add mock-RPC regressions for all emitted integer cells, schema metadata and
nullability, custom conversion hooks, and repeated warm construction.

Co-authored-by: Isaac <no-reply@databricks.com>

Build the subclass-label fallback fixture with an explicit object Index and
assert its concrete label type. Pandas 3 otherwise normalizes constructor
labels to builtin strings before the unchanged integer helper sees them.

The original regression passes on pandas 2.3.3 and fails on 3.1.0; the fixed
regression and full CI-selected Arrow/drop-in suites pass on both versions.
Runtime Python and native sources remain unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>

Decline noncanonical columns indexes and their direct string-storage hooks before the integer fast path inspects labels or normalizes cells. Keep native tolist errors and mutations in their original order and cover public construction on both pandas versions.

Co-authored-by: Isaac <no-reply@databricks.com>

Decline class-level columns storage descriptors before reading labels. Keep the single native columns storage read before pandas normalization, and cover public construction with inferred, named and explicit schemas on both pandas versions.

Co-authored-by: Isaac <no-reply@databricks.com>

Avoid loading optional Arrow label modules for fallback inputs, and decline
integer Arrow conversion when the loaded importer AtomicType alias is absent.

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Read the three mutable Arrow normalization settings in one Get request per nonempty construction while retaining empty-input and transport behavior.

Co-authored-by: Isaac <no-reply@databricks.com>

…rrow

Conservatively accept ordinary object-column frames with exact built-in dates and explicit DateType schemas. Preserve native row conversion for customized schema descriptors and unsupported values.

Co-authored-by: Isaac <no-reply@databricks.com>

Use the measured 10000-row threshold to avoid config RPC overhead on smaller inputs. Cover the fast-path boundary and load fresh-process test fixtures by exact source path for CI portability.

Co-authored-by: Isaac <no-reply@databricks.com>

Use the shared protocol fixture so fresh-process descriptor regressions skip consistently with the Arrow suite when protoc is unavailable.

Co-authored-by: Isaac <no-reply@databricks.com>

Require concrete pandas NumPy storage and unchanged converter aliases before selecting the date Arrow path. Preserve custom array values and native fallback error ordering.

Co-authored-by: Isaac <no-reply@databricks.com>

Keep astype, notna and where class replacements and instance hooks on the native row path. Authenticate ordinary pandas normalization methods before columnar date conversion.

Co-authored-by: Isaac <no-reply@databricks.com>

Decline customized column label storage before date column inspection so native row conversion retains hook values, exceptions, and normalization order.

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Build on the canonical pandas Index and integer guards from PR121.
Serialize eligible native float64 columns to inline Arrow IPC without row
materialization, preserving explicit schema and pandas missing values.

Include the shared internal inline IPC bridge and public Float coverage.

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

…h never reaches, and document the path

The float64 path builds its Arrow table and IPC itself, so the checks of the converters of
pyspark.sql.pandas.types and of the Arrow importer guarded code it does not call, and
imported pyspark.sql.pandas.types for nothing. Drop them, with the three tests that only
exercised them (they pass with either path), and describe the path in the architecture
docs.

Co-authored-by: Isaac <no-reply@databricks.com>

…th in the float64 test

As in the int64 and timestamp tests: a later row conversion of the same frame reads
Index._data fewer times than the first, as pandas caches some reads.

Co-authored-by: Isaac <no-reply@databricks.com>

…ings

The float64 branch (the replacement of apache#143, SPARK-60727) carries the merge with master,
which has apache#121 (SPARK-60676), and the same inline IPC bridge as this branch. Take its
versions of the files this branch shares with it, and add the string path after the
float64 one: the dispatcher hook, prepare_string_input and the string tests.

Co-authored-by: Isaac <no-reply@databricks.com>

…onverter checks the string path never reaches

_pandas_string_ipc encoded every string to UTF-8 in a validation pass and again while
building; build in one pass and fall back on the first string that cannot be encoded.
prepare_string_input calls the Rust encoder with the native schema, so the checks of the
aliases of the Arrow importer and of pyspark.sql.pandas.types guarded code it does not
call; drop them, with the two tests that only exercised them.

Co-authored-by: Isaac <no-reply@databricks.com>

…th in the string test

As in the int64, timestamp and float64 tests: a later row conversion of the same frame
reads Index._data fewer times than the first, as pandas caches some reads.

Co-authored-by: Isaac <no-reply@databricks.com>

…float64

Master has apache#130 (SPARK-60686), which adds its timestamp path at the same places: keep it,
and add the float64 hook after the timestamp one, and the float64 functions and tests
after the timestamp ones.

Co-authored-by: Isaac <no-reply@databricks.com>

…ings

Co-authored-by: Isaac <no-reply@databricks.com>

Stack this on the replacements of apache#143 and apache#144 (apache#207 and apache#208), on the current master,
which has apache#121, apache#128, apache#130 and apache#134 that this branch carried copies of: take the stacked
branch's files, and add the date hook after the string one, and the date functions and
tests after theirs.

Co-authored-by: Isaac <no-reply@databricks.com>

…ct date capture mock, and do not pin the reads of master's pandas row path

The explicit DateType pandas case of test_exact_date_ipc_calendar_range has more than
10,000 rows, so it now goes through the Arrow importer, which reads its settings with a
Config RPC that the capture mock did not serve. As in the other pandas tests, a later row
conversion of the same frame reads Index._data fewer times than the first.

Co-authored-by: Isaac <no-reply@databricks.com>